On the Effectiveness of Parameter-Efficient Fine-Tuning

Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new for each task. Currently, many research works propose only fine-tune small portion parameters while keeping most shared across different These methods achieve surprisingly good performance and are shown more stable than their corresponding fully fine-tuned counterparts. such kind still not well understood. Some natural questions arise: How does sparsity lead promising performance? Why models? choose tunable parameters? In this paper, we first categorize existing into random approaches, rule-based projection-based approaches based on how they which tune. Then, show that all actually sparse conduct novel theoretical analysis them. We indicate imposing regularization original by controlling upper bound stability. Such stability leads better generalization capability empirically observed lot recent works. Despite effectiveness grounded our theory, remains open problem parameters. do utilize task-specific data information suffer from projection discontinuity problem. To parameters, Second-order Approximation Method (SAM) approximates with analytically solvable optimization function. The determined directly optimizing approximation extensive experiments several experimental results proposed SAM outperforms strong baseline also verifies analysis. source code paper can obtained https://github.com/fuzihaofzh/AnalyzeParameterEff\/icientFinetune .